Robot system, robot control device and method for controlling robot

ABSTRACT

A robot system includes a robot having a movable section, an image capture unit provided on the movable section, an output unit that allows the image capture unit to capture a target object and a reference mark and outputs a captured image in which the reference mark is imaged as a locus image, an extraction unit that extracts the locus image from the captured image, an image acquisition unit that performs image transformation on the basis of the extracted locus image by using the point spread function so as to acquire an image after the transformation from the captured image, a computation unit that computes a position of the target object on the basis of the acquired image, and a control unit that controls the robot so as to move the movable section toward the target object in accordance with the computed position.

BACKGROUND

1. Technical Field

The present invention relates to a robot system, a robot control deviceand a method for controlling a robot.

2. Related Art

Multi-axis control robots for industrial use have been used in factoriesor manufacturing sites for automation or labor-saving. Such a robot forindustrial use has a digital camera functioning as eyes of a human. Therobot can autonomously move to a position where it can grip a targetobject by performing image-processing of images captured during movementand can autonomously pass over an obstacle during the movement. Inrecent years, particularly, it has been necessary to enhance accuracy inpositioning by means of a high resolution digital camera and to performcorrection of blurring in a captured image so as to deal with a highspeed operation by the robot for industrial use. In a blur correctionmethod as described above, information about movement of a moving objectis detected by means of an acceleration sensor provided on a movablesection of a robot for industrial use, a point spread function(hereinafter, referred to as PSF) indicative of a characteristic ofblurring due to the movement is obtained on the basis of the detectedinformation, and the correction of blurring is performed on the basis ofthe PSF. JP-A-2008-118644 and JP-A-2008-11424 are examples of therelated arts.

However, since the acceleration sensor is provided on the movablesection of the robot for industrial use, fluctuation in temperature orvariation in acceleration may be increased depending on a workingenvironment or work so that reliability or accuracy of the informationoutput from the acceleration sensor may be degraded. In addition, inorder to obtain the PSF with high accuracy, an image capturing timing ofthe digital camera and an output signal from the acceleration sensorhave to be synchronized with each other so that much cost or labor isrequired.

SUMMARY

The invention intends to solve at least part of the above problem, andcan be realized by the following aspects.

A robot system according to a first aspect of the invention includes arobot having a movable section, an image capture unit provided on themovable section, an output unit that allows the image capture unit tocapture a target object and a reference mark and outputs a capturedimage in which the reference mark is imaged as a locus image, anextraction unit that extracts the locus image from the captured image,an image acquisition unit that performs image transformation on thebasis of the extracted locus image by using the point spread function soas to acquire an image after the transformation from the captured image,a computation unit that computes a position of the target object on thebasis of the acquired image, and a control unit that controls the robotso as to move the movable section toward the target object in accordancewith the computed position.

With the above configuration, the locus image of the moved referencemark is extracted from the captured blurred image obtained by capturingthe target object together with the reference mark during movement ofthe movable section. While making the extracted locus image to be apoint spread function, an image whose blurring is reduced, is restoredfrom the captured blurred image by the image transformation by using thepoint spread function. A position of the target object as a movingdestination of the movable section is computed on the basis of with therestored image so that the robot is controlled so as to move the movablesection toward the target object in accordance with the computedposition. As a result, since the locus image of the reference markimaged in the captured image is made to be the point spread function(PSF), it is possible to obviate the need of a device for acquiringinformation about the movement and to facilitate the acquisition withoutdegrading the reliability and the accuracy.

In the robot system according to the first aspect of the invention, aperiphery of the reference mark is preferably a monochrome region havinga difference in brightness between the monochrome region and thereference mark so as to enable the reference mark to be identified asthe reference mark.

With the above configuration, the extraction unit can extract thereference mark by readily identifying the reference mark.

In the robot system according to the first aspect of the invention, thesize of the monochrome region is preferably greater than a size of ablurred region produced by movement of the movable section in thecaptured image.

With the above configuration, even when any other article image isimaged adjacent to the reference mark in the captured blurred image, theextraction unit can extract the reference mark by readily identifyingthe reference mark.

In the robot system according to the first aspect of the invention, therobot system further preferably includes a visual recognition unit thatallow a user to visually recognize the reference mark included in therestored image.

With the above configuration, a user can recognize whether or not theimage is adequately restored by visually recognizing the reference markincluded in the restored image.

The robot system according to the first aspect of the invention,preferably includes two or more reference marks whose respectivepositions are already known.

With the above configuration, detection can be performed even when themovable section moves straight or rotationally moves.

A robot control device according to a second aspect of the inventionincludes a robot having a movable section, an image capture unitprovided on the movable section, an input unit that receives a capturedimage including images of a target object and a reference mark, thereference mark being imaged as a locus image, an extraction unit thatextracts the locus image from the input captured image, an imageacquisition unit that performs image transformation on the basis of theextracted locus image by using the point spread function so as toacquire an image after the transformation from the captured image, acomputation unit that computes a position of the target object on thebasis of the acquired image, and a control unit that controls the robotso as to move the movable section toward the target object in accordancewith the computed position.

With the above configuration, the locus image of the moved referencemark is extracted from the captured blurred image obtained by capturingthe target object together with the reference mark during movement ofthe movable section. While making the extracted locus image to be apoint spread function, an image whose blurring is reduced, is restoredfrom the captured blurred image by the image transformation by using thepoint spread function. A position of the target object as a movingdestination of the movable section is computed on the basis of therestored image so that the robot is controlled so as to move the movablesection toward the target object in accordance with the computedposition. As a result, since the locus image of the reference markimaged in the captured image is made to be the point spread function, itis possible to obviate the need of a device for acquiring informationabout the movement and to facilitate the acquisition without degradingthe reliability and the accuracy.

A robot control method according to a third aspect of the invention isused for a system including a robot, a target object that is a movingdestination of a movable section provided on the robot, an image captureunit provided on the movable section, and a reference mark. The robotcontrol method includes (a) inputting a captured an image including bothimages of the target object and the reference mark captured by the imagecapture unit during movement of the movable section, the reference markbeing captured as a locus image, (b) extracting the locus image from theinput captured image, (c) restoring an image obtained from the capturedimage by performing image on the basis of the extracted locus imagetransformation by using the point spread function, (d) computing aposition of the target object on the basis of the restored image; and(e) controlling the robot so as to move the movable section toward thetarget object in accordance with the computed position.

With the above method, the locus image of the moved reference mark isextracted from the captured blurred image obtained by capturing thetarget object together with the reference mark during movement of themovable section. While making the extracted locus image to be the pointspread function, an image whose blurring is reduced, is restored fromthe captured blurred image by the image transformation by using thepoint spread function. A position of the target object as a movingdestination of the movable section is computed on the basis of therestored image so that the robot is controlled so as to move the movablesection toward the target object in accordance with the computedposition. As a result, since the locus image of the reference markimaged in the captured image is made to be the point spread function, itis possible to obviate the need of a device for acquiring informationabout the movement and to facilitate the acquisition without degradingthe reliability and the accuracy.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention will be described with reference to the accompanyingdrawings, wherein like numbers reference like elements.

FIG. 1 is a schematic view showing a robot system according to a firstembodiment of the invention.

FIG. 2 is a plan view showing a functional structure of the robot systemaccording to the first embodiment of the invention.

FIGS. 3A through 3E are explanatory views showing a captured marker anda captured reference pattern according to the embodiment of theinvention.

FIG. 4 is a flowchart showing a flow of processes of the robot systemaccording to the first embodiment of the invention.

FIGS. 5A and 5B are explanatory views showing arrangement of a markerand a reference pattern according to a second embodiment of theinvention.

FIGS. 6A and 6B are explanatory views showing rotational movement of themarker and the reference pattern according to the second embodiment ofthe invention.

DESCRIPTION OF EXEMPLARY EMBODIMENTS

The preferred embodiments according to the invention will be describedwith reference to the accompanying drawings. It should be noted thatembodiments described below do not limit the spirit or scope of theinvention defined by the appended claims and all the structures in theembodiments described below are not necessarily needed to form thesolution of the invention.

First Embodiment

FIG. 1 is a schematic view showing a robot system 5 according to a firstembodiment of the invention. The robot system 5 is equipped with a robot10 and a robot control device 50 that controls the robot 10. The robot10 is a six-axis control multi-joint type robot that has movablesections including a robot base 25 as a base body to be placed on aplacement face, a first shaft 11 that rotates around a vertical axiswith respect to the placement face, a second shaft 12 that turns an arm17A about a horizontal axis, a third shaft 13 that rotates in an axialdirection of the arm 17A, a fourth shaft 14 that is fixed to the arm 17Aand turns an arm 17B in a horizontal direction, a fifth shaft 15 thatrotates in an axial direction of the arm 17B, and a sixth shaft 16 thatis fixed to the arm 17B and turns a wrist 18 in the horizontaldirection. A hand 19 for gripping a component and a digital camera 20 asan image capture unit for capturing in a gripping direction of the hand19 by means of an imaging device such as a CCD (Charge Coupled Device,not shown) are attached to a tip portion of the wrist 18.

While the six-axis control multi-joint type robot is used as the robot10 in the first embodiment, it is not limited thereto and a scalar typerobot can be used. The use application of the robot 10 is not limited tothe gripping of a component by the hand 19, but can be gripping of atool for performing processing such as soldering or welding. Inaddition, the robot is not limited to one for industrial use, but formedical use or home use. The above driving shafts are configured so asto be rotated by driving of a plurality of actuators 30 (in FIG. 2)which are operated by a motor or an air compressing device (not shown).The plurality of actuators 30 are configured so as to be driven inaccordance with control signals transmitted via a cable 85 from a robotcontrol device 50. The digital camera 20 captures an image in apredetermined exposure time period, and the captured image istransmitted to the robot control device 50 via the cable 85.

A single plane marker 140 having a predetermined size and a monochromecolor region is disposed at a portion in the vicinity of a workpiece 130gripped by the hand 19 of the robot 10. A reference pattern 150 as apoint-like mark is in-printed on the marker 140 at a roughly centralpart. The hand 19 and the marker 140 are disposed so as to be capturedby the digital cameral 20 in an identical view field of the digitalcamera 20 and are formed to have colors and sizes so as to be identifiedwith each other when they are captured by the digital camera 20. In thefirst embodiment, the marker 140 may have a gray color monochrome regionand the reference pattern 150 may have a black color. However, themarker 140 and the reference pattern 150 are not limited thereto. Thereare no limitations on the color of the monochrome region as long as themonochrome region has a brightness level and a color saturation eachbeing in an extent facilitating identification of the reference pattern150.

The robot control device 50 is equipped with a computer 80, a display 82and a keyboard 84. The computer 80 is constituted by hardware resourcesincluding a CPU (Central Processing Unit), a RAM (Random Access Memory),a ROM (Read Only Memory), an HDD (Hard Disk Drive), a sequencer, a robotcontroller and a drive unit (each being not shown). The computer 80realizes functions of functional sections (described later) by aflexible association between the above hardware resources and softwarestored in the ROM or HDD.

FIG. 2 is a schematic block diagram showing a functional structure ofthe robot system 5. The robot control device 50 is equipped with apositional information acquiring section 100, a robot operation controlsection 115 and a driver section 120. The positional informationacquiring section 100 has an image input section 102, a referencepattern extracting section 104, an image restoring section 106, arestore recognizing section 108 and a position computing section 110,and acquires positional information of a workpiece 130. The functionalstructure of the robot system 5 is described below with reference toFIG. 2 and FIGS. 3A and 3B. The image input section 102 is an input unitthat receives an image signal output from the digital camera 20. Theimage signal input to the image input section 102 is a signal of animage of the workpiece 130 viewed from the upper portion as shown inFIG. 3A, and the image includes the reference pattern 150 and the marker140 in the identical view field. The image is transmitted to thereference pattern extracting section 104. Since the digital camera 20 isattached to the hand 19 of the robot 10, when capturing of an image isperformed by opening a shutter for a predetermined exposure time periodduring the movement of an arm 17, a blurred image due to the movement isoutput.

FIGS. 3B and 3C are schematic views showing movement of the referencepattern 150. Along with the movement of the arm 17, the marker 140 ismoved toward a right upper direction as shown by images 140A, 140B and140C of the marker 140 in FIG. 3B and is captured and the referencepatterns 150 is moved toward a right upper direction as shown by images150A, 150B and 150C of reference pattern 150 in FIGS. 3B and 3C. As aresult, a signal of a blurred image including a locus image 160indicative of a locus of the moved reference pattern 150 as shown inFIG. 3D is transmitted to the reference pattern extracting section 104.In this instance, the images 140A, 140B and 140C of the marker 140 havean identical background region.

The reference pattern extracting section 104 is an extraction unit thatextracts the locus image 160 of the reference pattern 150 by analyzingsignals of the input blurred images. Information about the extractedlocus image 160 is transmitted to the image restoring section 106. Inthis instance, since the locus image 160 and the marker 140 havedifferent brightness levels and color saturations, they can beidentified with each other in accordance with the differences so thatthe locus image 160 can be readily extracted. In the above case, themarker 140 can be first extracted from the blurred images on the basisof a color component or a shape of the marker 140. Next, the locus image160 included in the marker 140 can be extracted on the basis of adifference in the brightness level. In a case where a background articleis imaged in a background region where the images 140A, 140B and 140C ofthe marker 140 captured during the exposure time period are notsuperposed in an image captured by the digital camera 20, the locusimage 160 may not be possibly extracted.

Therefore, a size of the marker 140 is determined depending on theexposure time period of the digital camera 20 and a moving speed of thearm 17 so that an overlapped region of the marker 140 produced by themovement of the arm 17 in the blurred image includes the locus image160. Namely, the size is determined to be larger than a maximum blurredamount generated by the movement. In addition, the extracted locus image160 indicates information about the PSF for the spatial movement. Inthis instance, the brightness value of the locus image 160 varies inaccordance with the moving speed of the arm 17. Namely, the more themoving speed decreases, the more the brightness values of points formingthe locus image 160 increase, and vice versa. Consequently, in the firstembodiment, a change of the brightness value of the line-like locusimage 160 can be acquired within a range of assumed moving speeds.

The image restoring section 106 as a restore unit performs imagetransformation (inverse conversion) of a blurred image by using the PSFindicated by the locus image 160 extracted by the reference patternextracting section 104. By the above image transformation, the blurringis corrected so that an image having less blurring (a restored image)near an original image can be restored. Note that, the above method isdescribed in, for example, JP-A-2006-279807. The restored image whoseblurring is corrected is transmitted to the position computing section110. As a result of the correction of the blurring, the locus image 160involved in the image before correction is converted to a singlereference pattern 150 having no blurring in the restored image as shownin FIG. 3E.

The restore recognizing section 108 as a visual recognition unit allowsa user to visually recognize that the correction is adequately performedby displaying the converted reference pattern 150 on the display 82.Here, when the user recognizes that the reference pattern 150 is notadequately restored as a result of the visual recognition, the user caninstruct the restoring of the image again to the positional informationacquiring section 100. The position computing section 110 as acomputation unit computes a position of the workpiece 130 in accordancewith the restored image and transmits information about the computedposition to the robot operation control section 115. The robot operationcontrol section 115 and the driver section 120 constitute a control unitthat controls an operation of a movable portion of the arm 17. The robotcontrol section 115 computes a driving amount of each of the pluralityof actuators 30 for moving the robot 10 in accordance with the receivedinformation about the position and transmits the driving informationabout the computed driving amount to the driver section 120. The driversection 120 generates a drive signal for each of the actuators 30 inaccordance with the driving information received from the robot controlsection 115 and transmits the drive signals to the respective actuators30. As a result, the arm 17 may move to a predetermined position or thehand 19 may grip the workpiece 130.

FIG. 4 is a flowchart showing a flow of processing in which the robot 10grips the workpiece 130 in the robot system 5. In the processing, ablurred image whose blurring is due to movement of the robot 10 is firstinput (step S200). Next, the CPU of the computer 80 extracts a PSF as areference pattern in the input image (step S202). Then, the CPU restoresthe image whose blurring is corrected in accordance with the extractedPSF (step S204). Next, it is determined whether or not the image isadequately restored (step S206).

When it is judged that the image is not adequately restored (“No” instep S206), the CPU returns to the process (the step S202) in which thePSF as the reference pattern is extracted from the input image andrestores the image again (step S208). On the other hand, when it isjudged that the image is adequately restored (“Yes” in step S206), theCPU computes the position of the workpiece on the basis of the restoredimage (step S208). Next, the CPU instructs the robot 10 to move to aposition capable of gripping the workpiece 130 and to grip the workpiece130 (step S210), and then finishes the series of processes.

With the above processes, in a case where the input image includes theblurring generated due to movement of the digital camera 20 attached tothe arm 17 during the capturing of the image, the image is subjected tothe inverse-conversion on the basis of the PSF indicated by the locusimage 160 of the reference pattern 150 on the marker 140 so as to obtainthe image whose burring is corrected. Further, the position of theworkpiece 130 is computed on the basis of the converted image and thearm 17 is moved to the computed position. As a result, the robot controldevice 50 can accurately acquire the position of the workpiece 130 andthe robot 10 can surely grip the workpiece 130.

Second Embodiment

A second embodiment of the invention will be described with reference toFIGS. 5A and 5B. Note that, the sections, portions or members the sameas in the above described first embodiment are denoted by the samenumerals, and their descriptions are omitted. In the first embodiment,the workpiece 130 and one marker 140 are disposed in the view field ofthe digital camera 20. However, in the second embodiment, the workpiece130 and four markers 140 whose positional relationships are alreadyknown are placed on an identical plane of the workpiece 130 at aperipheral portion of the workpiece 130 as shown in FIG. 5A. Note that,it is enough to put at least three markers 140 in the identical viewfield of the digital camera 20.

FIG. 5B is an explanatory view showing an image captured when the arm 17moves straight. While the image of the marker 140 is actually blurred inthe moving direction, the blurring of the marker 140 is not indicatedbecause its blurring is not essential for explanation of this case. Inthe case of the above described straight movement, the blurring due tothe movement can be corrected by processes similar to the firstembodiment.

FIG. 6A is an explanatory view showing an image captured when the arm 17rotationally moves around a position in the vicinity of the workpiece130. In this case, the robot control device 50 can obtain a rotationcenter and a moving direction as a coefficient of an affinetransformation in accordance with moving amounts of three or moremarkers 140 as shown in FIG. 6B. Note that, the affine transformation isa well known transformation method for expanding, contracting, rotatingor parallel moving of an image. As a result of the affinetransformation, even in a case where the arm 17 moves straight orrotationally moves or performs an operation with scaling, the robotcontrol device 50 can adequately restore an image from a blurred imageas long as the digital camera 20 moves without rotating around its axis.In addition, in a case where four or more markers 140 are input in anidentical view field of the digital camera 20, a change with respect toprojection transformation can be detected. In the second embodiment, itis possible to achieve an effect similar to that in the firstembodiment.

While the embodiments of the invention are described with reference tothe drawings, the specific structure is not limited thereto, andvariations or modifications in designing can be made without departingfrom the scope and spirit of the invention. For example, the marker 140can be made to have a black color, and the brightness value of thereference pattern 150 can be increased. The marker 140 is not limited toa plate shape, but can have a solid shape.

The entire disclosure of Japanese Patent Application No. 2008-286224filed Nov. 7, 2008 is expressly incorporated by reference herein.

1. A robot system, comprising: a robot having a movable section; animage capture unit provided on the movable section; an output unit thatallows the image capture unit to capture a target object and a referencemark and outputs a captured image in which the reference mark is imagedas a locus image; an extraction unit that extracts the locus image fromthe captured image; an image acquisition unit that performs imagetransformation on the basis of the extracted locus image by using thepoint spread function so as to acquire an image after the transformationfrom the captured image; a computation unit that computes a position ofthe target object on the basis of the acquired image; and a control unitthat controls the robot so as to move the movable section toward thetarget object in accordance with the computed position.
 2. The robotsystem according to claim 1 wherein, a periphery of the reference markis a monochrome region having a difference in brightness between themonochrome region and the reference mark so as to enable the referencemark to be identified as the reference mark.
 3. The robot systemaccording to claim 1 wherein, the monochrome region has a size greaterthan a size of a movement region produced by movement of the movablesection in the captured image.
 4. The robot system according to claim 1,further comprising: a visual recognition unit that allows the referencemark included in the acquired image to be visually recognized.
 5. Therobot system according to claim 1 comprising: two or more referencemarks whose respective positions are already known.
 6. A robot controldevice, comprising: a robot having a movable section; an image captureunit provided on the movable section; an input unit that receives acaptured image including images of a target object and a reference mark,the reference mark being imaged as a locus image; an extraction unitthat extracts the locus image from the received captured image; an imageacquisition unit that performs image transformation on the basis of theextracted locus image by using the point spread function so as toacquire an image after the transformation from the captured image; acomputation unit that computes a position of the target object on thebasis of the acquired image; and a control unit that controls the robotso as to move the movable section toward the target object in accordancewith the computed position.
 7. A robot control method for a systemincluding a robot, a target object that is a moving destination of amovable section provided on the robot, an image capture unit provided onthe movable section, and a reference mark, the robot control methodcomprising: (a) inputting a captured an image including both images ofthe target object and the reference mark captured by the image captureunit during movement of the movable section, the reference mark beingimaged as a locus image; (b) extracting the locus image from the inputcaptured image; (c) restoring an image obtained from the captured imagewhile and performing image transformation on the basis of the extractedlocus image by using the point spread function; (d) computing a positionof the target object on the basis of the restored image; and (e)controlling the robot so as to move the movable section toward thetarget object in accordance with the computed position.